Here are the benchmark results by timing the ./validate --partial-hackage-tests test.

• baseline (master branch)

  $ hyperfine --setup "./validate.sh" "./validate.sh --partial-hackage-tests"
  Benchmark 1: ./validate.sh --partial-hackage-tests
    Time (mean ± σ):     224.440 s ±  5.713 s    [User: 191.896 s, System: 34.800 s]
    Range (min … max):   218.596 s … 239.915 s    10 runs
  

• no-flatten-package-description

  $ hyperfine --setup "./validate.sh" "./validate.sh --partial-hackage-tests"
  Benchmark 1: ./validate.sh --partial-hackage-tests
    Time (mean ± σ):     232.745 s ± 10.650 s    [User: 194.793 s, System: 38.621 s]
    Range (min … max):   220.815 s … 254.096 s    10 runs
  

I'm not sure why validate is used for benchmarking parser changes. Validate is likely dominated by completely unrelated factors (e. g. solver). Would it be possible to just download a bunch of cabal files from Hackage (see clc-stackage for a curated set of packages) and run only the parser on them?

@ulysses4ever

Would it be possible to just download a bunch of cabal files from Hackage (see clc-stackage for a curated set of packages) and run only the parser on them?

The options --{partial,complete}-hackage-tests call the hackage test. These tests runs the readField and parsec parser against entire hackage. I'll look into clc-hackage as well, but it seems like hackage-tests is already exactly what we want here?

Validate is likely dominated by completely unrelated factors (e. g. solver)

I thought it would be important to test out whether I broke the solver by changing the representation to GPD too, which is why I tested everything :)

The options --{partial,complete}-hackage-tests call the hackage test.

these are tests, which is different from benchmarking. You seem to be using validate for benchmarking, which makes no sense to me, as explained above. On the other hand, using validate for tests may have some value although it is preferable to use the actual test like hackage-tests, which you do, so I don't have any issue with your approach to testing.

Would it be possible to just download a bunch of cabal files from Hackage (see clc-stackage for a curated set of packages) and run only the parser on them?

This is pretty much what the hackage-test test does. Do you think timing this alone would be more acceptable (compare to ./validate) even though it's not a benchmark?

If timing hackage-tests would mean timing download times too, then no, it doesn't sound right. I don't know if you can arrange for separate download, which would make it more reasonable. Generally, if you want to have an estimate for performance footprint, you have to develop a benchmark, you can't just hijack test executables whole-sale. You could probably reuse some of the testing infrastructure (tasty-bench exists for a reason). If this sounds too much, you should just stop trying to achieve it because currently you're posting numbers for validate that most likely have no connection to the actual performance change, and as such are actively misleading. It'd be great to have some very simple benchmark that shows that we don't degrade performance significantly but other than that we should focus on design rather than performance, I think: the design issue has been a die-hard for a long time...

If timing hackage-tests would mean timing download times too, then no, it doesn't sound right. I don't know if you can arrange for separate download, which would make it more reasonable.

hackage-tests looks for $XDG_CACHE/cabal/packages/hackage.haskell.org/01-index.tar and reads from it. It doesn't do the downloading (it would fail if the file doesn't exist and tell the user to run cabal update).

If this sounds too much, you should just stop trying to achieve it because currently you're posting numbers for validate that most likely have no connection to the actual performance change, and as such are actively misleading.

I totally understand. I can make sure cabal update is called before I invoke the test suite, does this make it more reasonable? It seems like the hackage test suite is doing what we want, which is parsing a huge collection of cabal files. I can also specify it to only use the read-fields (envelope cabal format) parser or the parsec (the actual cabal grammar) parser. If not, I'll not test the performance with hackage-tests anymore 👍

Otherwise, do you have an example of using tasty-bench to benchmark something similar in cabal? We wanted to add benchmark because in the tech proposal phase people were concerned about performance.